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Just as sociology at 'some point became caught up in the exercise 

> 

of creating grand theorie to the exclusion of grounded theory (Glaser 
and Strauss , 1967) , so education and other social action research be- 
came .enamored of creating research or evaluation dat^ de novo : new 
questionnaires, new tests, new sets of interview protocols, new inter- 
views (often of the same persons), all related to a priori concepts. 
Equally often, in the process of generating such de novo findings other 
valuable nxat^rials were overlooked, unnoticed, or scorned because of 

♦ 

their very availability or "joutineness" . 

^^ut the cost of research has skyrocketed. Personnel, time, trav- 
el,' equipment, sites, subjects, research assistants, computer techni- 

V 

cians and compute/ processing — all have contributed to making the 
enterprise of research costly, time-consuming, and labor intensive. In 
the face jbf Ms^g cost^ hard questions are being raised about the 
Vcilue of suth research, especially jvhen the resulta*#produce data which, 
conclude that there are "no significant differei?:^s" , when the overall 
body of accumulated knowledge has not been well > informed, or when 

« . * 

description provided has proven inadequate or misleading to other 
inqui^^s . . Senator P^oxmire's monthly Golden Fleece awards are h\\t an 

amusing, if embarr^sing, tip of the iceberg .when the! subject under/ 

\ / 
consideration is waste in sponsored research and evaluation (especially 

/ 

Federally funded efforts). Hard questions are being as^ed from several 

quarters about the' value of inquiry whicl^ turns up little new thfornia-, 

tlon or few results which inform public pofticy. ^ 

Undoubtedly much of the error 'lies in method. But some surely 

« 

may be attributed to the- fact that as social inquirers, it has appeared 
more •prestigious to us to generate "new" information than to resort to 



other activities which are promising but hardly lend "star status" to 
our careers. We refer to reanalysis of previously collected data; to 
meta-analysis of data (i.e., cli^ster' analysis, case study aggregation ; 
analysis or propositional analysis): and to secondary analyses (including 
documentary analysis--such as content anal:^.sis— a^d record analysis — 
including trend and aggregation analysis). 

Documents and records are usually readily available, often public, 
non-reactive and unobtrusive, and why educational inquirers would not 
want to use them is unclear . ^ Records particularly are an enormously 
useful spurce, although other documents, such as evaluation reports, 
technical reports, case studies and the like, are now amenable to such 
\ analyses that not only aggregate common information, but lead to new. 
Insights into public policyfand its formulation (Lucas, 1974a, b; Lincoln, 
1977). But educational inquirers appear to be bent on generating de 
novo data for every problem they attack, perhaps because they tyi3jl^l-' 
ly desire to test an hypothesis ^d such tehts seem to calV for new 
instruments or special dat^a collection arrangements. If Denny (1978, p. 
21) is Correct that, "If the contributions of educational psychology and 
evaluative research to our understanding of teaching ani learning could 
be translated into human stature it would stand a little over four feet 
high," then a new approach is surely needed. Jt is, ^oincidentally, 
^probably also true that the consistent failure to utilize documents and 
*^ records (at least in fair proportion to their existence and availability) 
accounts in part for the fact that educational inquiry inputs so often 
are"' not grounded. If inquirers had to discipline themselves to n^iake 
their experimental situations and their findings mak6 sense in terms of 
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context^ their results might be more meaningful, rather than the "dreary 
palaver we call research and evaluation literature" (Denny, 1978, p. 



22). More will )je said about the importance of context shortly. 

This chapter will deal with both documents and records, their 
acquisition, aggregation and analysis. Though both are (usually) 
writter/ records (occasionally , they are . verbal transciptions , films or 
photographs) , a document is not the same , thing as a record.. . Never- 
theless, both are forms of communication based on verbal behavior. If 
"The wori of the world, and its entertainment, is in no small measure 

■ K 

mediated by verbal and other symibDlic behavior," (Cartwright, 1953, 
p. 422) then making senge of the records of verbal behavior left is a, 
crucial part of the investigation of man and his social behavior. 

What is a document? What is a record? 

i 

Neither the dictionary nor the methodological literature is of much 
assistance in making a distinction between documents and records. The 
dictionary, for example,^ defines a document as "an original or official 
printed or written paper furnishing information or used as proof of 
something else/' and a regard as "an official written account of pro- 
ceedings" 9r "the known facts regarding someone." The overlap be- 
tween these definitions is obvious; indeed, following lead of the 

« 

dictionary one might well define a document as a record and a record ^s 
a document. The#methodol6gical literature confounds this problem even 
further by admimng as documents a variety of written materials that 
are projluced only because an inquiry has been undertaken. *Thus 



\. 

HplsU (1969) Indicates that documents include "verbal data produced by 

f 

subjects fl|t the. behest of the investigator, ,, .the psychiatric interview, 
and ^various projective instruments such as the Thematic Apperception 
Test. [and] responses to open-ended questions generated in survey 
research, .. .written messages, derived from a simulatipn study,... or 
communication produced during group interaction." Bodgan and Taylor 
(1975> -and others suggest similan definitions^ While it is true that 
written residues of these kinds can be analyzed, by the methods to be 
described in this paper, we believe it is useful for analytic and peda- 
gogic purposes to exclude them from consideration here. Hence we 
shall define a record as a written statement prepared by an ij[;Kiividual 
or an agency for the jpurpose of attesting tq an event or providing an 
accounting, and a ddcument as any wjritten (or filmed) material other 
than a record and which was not prepared specifically in response to 

I 

some request from or some task set by the investigator. 

What is an example of a document? A list of examples of docu-, 
ments could, with some imagination and creativity, go on for pages: 
^t good examples of documents include: letters, memoirs, ^lutojpio- 
graphies , / diaries or journals , textbooks , wills , position papers , suicide 
notes, speeches, novels, newspaper articles and editorials, epftaphs, 
television .and film scripts, memoranda, case studies^ life histories, 
medical histories, political pr^aganda pamphlets, government publi- 
cations, photographs, diplomatic communiflues, and the like. 

* 

Documents can be sorted into various typologies. The most ob- 
vious category is the source of the document. Another very useful dis- 
tinction between '"primary and secondary" documents, the latter falling 



Into the class of what would be called ^^^^arsay" in a court of law. 
That is to say, it is likely not generated from firsthand experience of a 
situation or event, but rather from other sources. Other useful dichot- 
omles for sorting documents include those of "solicited" vs. "unso- 
licited," that is, requested by someone (usually the inquirer) versus 
volunteered; "comprehensive" vs. "limited," "edited" vs. "complete" or 
"unedited;" and "anonymous" vs. "signed" or attributable (Bogdan and 
Taylor, 1975, p. 96).^ To these we would also add another distinction, 
that between "spontaneous" (as in a. diary) ^xvd "intentional" (as in a 
letter to an editor) , although obviously in those categories "spon- 
taneous" may corcie clojse to being "unsolicited" or I'unedited" and "inten- 
tiohkl" may closely resemble also "edited" and "signed." 

The very number of ^iiiestf^ typologies make the whole matter of 
documentary classification very complex. If we use the six dichotomies 
suggested above, we have to contemplate 2 ^=64 categories. These 
sixty-four categories may be further Subdivided in terms of the' appar- 
ent motivation of the writer. If for example it was decided to use a 
simple five-category motivational system — explication, support, self- 
Justification, moral duty and self-aggrandizement— we could conceivably 
have enlarged our taxonomy to 64 x 5 = 320 categories. Since the 
prospect of dealing with that number of possible cells (or more) boggles 
the Boind, the bases for categorization are probably more usefully view- 
ed as criteria for judging the utility of a document rather than for 
assigning it unequivocally to some cell of known properties within a 
taxonomy. 

What is an example of a record? A record as noted before is a 
written, residue attesting to an event or situation, or providing an 



accounting of an event or transaction. Records have typically not been 
utilized in educational research or evaluation, as indeed they have not 
in many forms of social research, with the notable exceptions of demog- 

I 

raphy and cliometrics, which has leaned on them rather heavily. Exam- 
ples of records include: airline manifests, audits and consultants' re- 
ports, birth records, business records, cab campany records, campaign 
contribution and expenditure records , charitable gifts records, chatUe 
mortgage records, city directories*, death records, expense account 
vouchers , f ina^icial information , government directories , gun registra- 
tions, income lax records, private organization records and membership 
lists, legal newspapers, legal notices, marriage records, military rec- 
ords, professional, business and trade directories, religious directories, 
school directories ,\ Securities and Exchange C9mmision records, state 
regulatory records, telephone company records, vanity records, vehicle 
records, voting records and. registration, welfare records/ zoning, land 
use and property tax records, county plat books, court records, and 
police records and election records. The list of types of records is 
clearly and apparently limitless, and with computer technology revolu- 
tionizing record storage^nd retrieval, it can be reasonably expected 
that the list of types of records kept (both knowo and unknown) will 
grow exponentially over the next few years. 

Unlike documents , whch come into being for a variety of motiva- 
tional purposes, ritords. are generally compiled simply to "keep track" 

« 

of events or transactions. ' They form an official statement that some 
event or transaction has *(iccurred, and while they may be altered or 
forged in some way, much of the time they are public, reasonably 
direct, and reasonably accurate (accessibility to an investigator, how- 



ever, may be limited, since some records--such as bank records and 
income tax returns— are ^enet^lly not available to those who are not 
authorized to review them). ' / 

Why differentiate between records and documents? There are two 
major reasons why distinctions are made between documents and records • 
-The first Is Uiat they represent different motivations or purposes on 
the part of the writer. Records, as has been noted,. attest to an event 
or transaction, and form an official chronicle which is part of an larger 
work, usually on the processes and proceedings of public affairs. 
Documents on the other hand, may be personal (private) or public If 
public, they serve to make others aware of a point of view, persuade, 
aggrandize, explicate or justify. If personal, they may be a form of 
special pleading (e.g., a letter) or exhibitionism, they may arise from a 

\ 

desire for order or a relief from tension, they may serve as therapy, 
literary delight, or public service, they may spring from the desire for 
monetary gain (witness the Howard Hughes "autobiographies"), they 
may be assigned (e^g., one may be instructed to create a memorandum 
yhich forms a 'letter of understanding'), they may be writtfen for 
scientific interest (e.g., the observation logs o/ Thomas Jefferson and 
Benjamin Franklin), or they may serve as social reincorporation devices 
as did the Nixon interviews with Dav|d Frost (Allport, 1947, pp. 68- 
75). Thus records and documents ar^^ntirely different forms of writ- 
ten or verbal records, arising *ippm different motivations and serving 
separate purposes. 

A second cukTrnuch more important reason to differentiate carefully 
between documents and records is that the modes of .analysis appropri- 
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ate to each are substantially j|ifferent. In the case of records, appro- 
priate analytic tools include aggregation/ integration methods, trend 
analysis and what we shall term "tracking." The former two are essen- 
tially quantitative methods and hence will not be discussed here. In 



ysis and the case survey aggregation method, both of which shall be 
considered. The differentiation between documents and records both in 
respect to definition and appropriate modes of analysis is illustrated in 
Figure 1. • 



Some social scientists have never questioned the value of records 
and documents for inquiry. They have utilized them extensively to 
predict and explicate fertility patterns , population shifts , needs an.d 
desires for housing, transportation and consumer goods, and for pat- 
terns of education. Likewise, an investigative journalist learns early on 
that records and documents are invaluable resources for tracing trans- 
actions which may shed light on cases and relationships under scrutiny. 
Hage, et al (1976, p. 41) caution that, "Investigative reporting is not 
often as exciting and glamorous as that of Woodward and Bernstein., ~ 
More often it involved laborious checking of public records, finding 
documentation for the story." Williams likewise asserts (p. 37, 1978), 
"The first and great commandment of investigative journalism is this: 
get the record.^* 

If other types of inquiry have relied heavily* on documents and 
records, wfiy has not educational inquiry? Perhaps this failure relates 
to the earlier • arguments for. de novo data collection. As likely, it 



the case of documents, appropriate 
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-Figure 1. Distinctions Between Documents and Records 







' Purpose 


Analytic Tools 


Documents 


Any written residue 
other than a record 
and not produced by 
the investigator. 


Content analysis 
Case survey method 


Records 


Any written residue 
attesting to an 
event or providing 
an accounting. 


Aggregation/integration 
Trend analysis 
Tracking 



SOME DIFFERENCES BETWEEN DOCUMENTS AND RECORDS 



2tes to the lack of training in , the use dl records and documents, to 
relative ease with whiclWmost of them are acquiued, with its attend- 
ant assumptions about familiarity and contempt, or to the general lack 
of ' regard for context in which educational decisions dre made, programs 
are carried out, and policy is constructed. For whatever the reason, 
inquirers cannot much longer afford to ignore the resource. 

First, documents and records, are a stable, rich and rewarding 

\ 

resource. Both tend to persist; that is, \yhile they may be burled back 
in files which are no longer used, they are often available for the 
asking. They provide a base from which any subsequent Inquirers can 
work, and thus lend stability to further inquiry. 

-Second, records (as opposed to documents, although occasionally 
the same WilP hold true for those writings also) constitif te ,a legally un:: 
assailable base from which to defend oneself against a^egations, inter- 
pretations, and libel. The best defense in a challenge to an evaluation 
report is to be able to show that you told only the truth, and the best 
evidence for truth is most often the public record. 

•-Third, both documents and records represent a "natural", source 
of information— a delight to the naturalistic inquirer. Not onjly are 
they, in fact, an "in context" source of information — that is, they afise 
frOm the context and exi^t in it — but they consist of information about 
the context. That is, records record what happened in the context, 
and documents record a variety of other evidence about the environment 
and people's perceptions of it. They are repositories, as a result, of 
some of the best grounded data available on the events or situations 
under Investlgatioh. 
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Fourth, they .are available on a Ipw-cosf or no co^t basis, re- 
quiring often only the Investigator's time and energy. Public records, 
particularly, are readily available and for the most part, open to scru- 
tiny. Although documents may exist corporate or project files, they 
also might not, and tracking them down might involve some, ingenuity. 
For the naturalistic inquirer, the obvious places fo search come first, 

-the ingenuity later. The most apparent sources qf documents are old 
files and persons who were associated with the project or program. 
But often, files are stored and cannot be easily* retrieved, and persons 
have moved on to othfer sites, died, retired, or are otherwise unreach- 
able. A next move wp.uld be to try other persons not as closely related 
to the project, but who have had some connection with it. Those 
persons will often collect random documents from the cif filiated project, 
and maf have them. Failing easily reachable sou;^ces, the document 
hunter moves in ways which resemble either the historian searching for 
written- evidence (Altick, 1950) or the interviewer attempting to infler- 
view someone who does not wish to be interviewed (Dexter, 1967^. If 
some source h,as a document, or is thought- to have some document 
which Is particularly important, often persistent courtesy (with empha- 
sis on both persistent and courtesy) will prevail. . Sources of documents 
and records can often be persuaded to part with them simply because it 
Is finally easier to get rid of an inquirer than to keep saying no. Most 
possessors of documents do^ not allow access for one of four reasons: 
Indifference, hostility, ignorance, or finally > avarice (Altick, 1950, p. 

^16). Nearly " all of those reasons can be overcome by a final tactic, 
that of the introjiuction by 'a friend or a^^^|||te of the family or '<j[ot3^ 
ment holder. Introduction by trusted famUiars^as long been us^d to' 
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make social connections. It is an equally worthy technique for those 
desiring interVi^^ or seeking missing documelj|ts, and ought not to be 
overlooked |n the effort to prevail upon a^cument holders to release 
materials\ 

Fifth, documents and records both' are i^qn-reactive. Although 
there are instances when access to primary persons is impossible in a^iy 
event (the person has died, for instance) and the only remaining way 
.to study him or his connections is through documentary analysiis, there 
are also other times when, althoigh a person is available, documents 
still . provide the most objective^ ack for understanding some aspect of 
his performance or behavior, ^l^lsti (1969, p. 15ff) addressed this 
problem thus: . 

"Despite their very re^l merits for social research, even the best 
experiment or survey studies the subject and his responses in a 
highly artificial situation. Knowledge that one is being studied 
may, in some circumstances, materially alter those -^spects of 
behavior under analysis. Especially when it is importanl-to get 
repeated measures qif the subjects' values, attitudes and the like 
over a period of time, and if one has reason to believe that con- 
tinued Interaction betWeen the investigator and subject may alter 
behavior, then analysis of the subject's statements may be a useful 
way to gather the required data. An important feature of content 
analysis Is that It Is a "nonreactive" or "unobtrusive" research 
technique." ^ 

Thus , when live Interaction with a. subject would likely alter 

* V t • ♦ 

behavior or perspectives, then . docunjentary analy^s (specifically, 
content analysis) is a useful way' of gathering information for analysis 
which is not subject to Interaction effects between collectpr and subject. 

Sixth, whether or riot the inquirer finally decides* to interact with 
his subject(s), content analysis 'and other forms of documentary and 
r€(Cord analysis enable supplementary and contextual data tso be gather- 
ed. That Is, documentary analysis may be an extension of a larger 
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body of research, an additional technique employed, or the primary 
form of research itself. In any event, it lends contextual wealth and 
texture, and grounds any inquiry in the milieu of the writer. The 
grounding in real-world issues and day-to-day concerns is ultimately 
what the naturalistic inquirer is about. 

Within the framework of historiography, the art and Science of 
writing history, the importance of context has been asserted continual- 
ly. Clark's injunction to writers of hisl^y still holds true: 

"The most important principle of historical scholafship is 
the principle of importance of context . When considering 
any historical evidence, an investigator must take account 
of the situation at the moment in time when the event it 
records happened 3nd ^Iso at the moment when it was re- 
corded." (1967., p. 25, italics added). 

f 

pven earlier than Clark, Gray and others warned document analysts to 
V^eware of that worst of judicial -sins-^-taking the evidence away from 
its context." (1964, p. 58) ^ 

Within the framework of education research and evaluation, docu- 
mentary, and record analysis serves an opposite function; it fosters the 

> ■ 1 ' 

maintenance of iiiquirer interest in the context, and h^lps to insure 
that research is not removed from its social, historical and, political 
frame of reference • 

On the other side of the analytic fence, AUport, acting as deyiljs 
advocate for^* and against documentary analysis^ met some of the iCr^ti- 
cism against the use of personal documents in social science research. 
Some of those criticisms he found "to be well-grounded, other to befi 
irrelevant or trivial, and still others to be contingent upon the type of 
document employed and the use to which it is put" (1947, p. 125). 
Because the criticisms against document usage contain warnings about 
how they — and records — ought to be used, some review of the case 
against them as perceived by AUport appears in order. 
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There is' first the unrep.resehtativeness of the sample.^ In person- 

al docianents, this is particularly true, but it is also occasionally true 

in social actipti programs: often no one on the project ^keeps very good 
w / 

notes on yt)rocesses, few memoranda are generated, and even more 
often, only writing that is done is in response to funders* requests 
for technical reports or other periodic statements about program or 
WTdject progress. If no docimients exist, however, or if the documents 
are sparse and seem uninformative, this ought to tell "the inquirer 
something a^out the cbntext. Absence or paucity of documents relative 
to a project can bc| as commanding a statement as a careful content 
analysis, if the overall situation or environment is v^hat is to be des- 
cribed. , 

Second, there are certain stylistic and nonobjectivity criticisms of 
documents. AUport answers: 

"Since personal documents are, and always will be, complet-e- 
ly' s^vjajective , there is no Way of convincing the bitter-end 
objectivist that he should employ them. ..[But] (1) Extreme 
objectivism has -disclosed its own weakness. The resurgence 
iof phenomeh6logy has brought back to favor the personal 
repoik. (2) The conflict is hot so iri^concilable as it appears. 
Users* of. the pefsonal document have Aarned many lessons .from 
behavitM-isij^ai^ positivism.,." (1947, p. 127) 

To the \ist of criticisms may alsq be added concerns for the valid- 
ity of the doRun^nt. More will be said about this later in the chapter, 
but Allport maintains that there are at least three tests that one can 

us,e in establishing nonquantitative validity measures : "the general 

f 

honesty and credibility of the report. . .this is the ad hominen test. . . 

This refers only to oersonal documents and documentary evidences, 
not, ^ this instancer-to the use of Records. \^ 



The plausibility of the document in terms of our own Pfist 'experiences , 
as they are relevant, can be considered. .. [and] the test of Internal 
consfs tency or ' self-confrontation has-' to be widely relied upon." A 
document that hangs together, that represents a strjabtured config-' 
uration of human life and harbors no impossible contradictions has at 
least a prima facie validity" (p. 128). , 

There are problems of deception (deliberate), self Reception (which 

Allport calls "unintentional self-justification") and blindness to motives, 
which Allport ascribes to either the difficulty "for men to report ade- 
quately their own motives, and...tru^ wh^ther^ or not the mechanism of 
self -justification is at work" or to thte . "lapse of time that occuf^ be- * 
tween the completion of an act and its ifecording" (p. 132). Addition- 
ally, there are problems of oversimplificauon, wherein the writer "does 
not want unsolved riddles, visible gapsMunexpla^ned conduct. His 
desires for completeness lead him to fill in unknown parts in a manner 
that fabricates a satisfying closure." But, NA^^ort cautioned, "... 
third-person documents hkewise simplify, and. \. laboratory ai;id field 
investigations do the ^/Sme thing" (pp. 134-5). In iSmt case, the docu- 
ment analyst need simply be prepared to understand that programs, 
projects and persons are not as . straightforward and cbherent as" they 
are likely to be^ pictured, either by themselves or others. (To some 
extent, this is responsive to the later pleas of Wax tl971] ai\d Reinharz 
(19?8], to note, along with descriptions of research, the changes which 
occurred on site and in the inquirer.) Likewise, the effects of ..mood 
or errors of memory may« impinge on the accuracy (isomorphism, or 
one-to-one correspondence with the eve^t) or completeness '^^aps in • 
memory) of the document. In the Wiographic usage of person5J.-<locu- 



16 

ments, Allport steadfastly maintains mnemonic errors are significant as 

what the individual recalls and records, simply becausjH»"the very fact 

that the subjeet structures and recalls his life in a' certain manner is 

what we want to know" (p. 136). ' 

This last two criticisms are th^ Implicit conceptualizations (inherent 

In the writer's choice of theme and phrase) and the arbitrariness of 

conceptualization. The implicit cQticeptualizations— or the limitation of 

the author to ;data that* he thinks are Important- -is, in and of itself 

* 

useful, Allport points out, ami the arbitrariness of that same conceptu- 
alization, especially "when theN^erpretatlon is given on the ^tksis of 
manifestly meager data" (p. 139) is ho worse than what psydhologists 
and other social scientists already do to more "experimental facts*" He* 
concludes rather vw^ly: "That personal documents actually are any 
worse off <at the hands of psychologists than are other forms of raw 
data seems doubtful" (p. 139). So it would seem that so long as one 
operates .with a set t)f caveats — of which the above criticisms are but a 
part— the use of documents (especially) and ^records (to somewhat less 
extent) is legitimate, either as a primary or secondary (and supple- 
mwitary) technique. As ever, the guiding rule for choice of a method 

Is which of 'the techniques available provides more data, better data, 

/ 

and a lower cost than other methods. Further, those sources which are 
replete with clues as to the nature of the context should never be 
Ignored, whatever other inquiry methods one chooses. Although occas- 
lonally documentary or record analysis will be the singular research 
method . employed , it ought never to be ignored even when other field 
and research tecl^yiques form the primary information-gathering mode. 
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Methods and P roced ure s for p al in g with Documentfi and Record 



Although documents and records fall into a class of research 
evidences (verbal and primarily written), their acquisition, treatment 
and analyses are very different. For that reason, we shall treat those 
topic In separate sections which follow. 
Documentary Analysis . ^ 

Documents, as noted previously, are distinct from records in 
several ways. Persons interested in using records are primarily inter- 
V ested in "tracking," that is, in following official recordings of trans- 
ac^||is or Events, and perhaps determining the frequency or serialness 
of events or transactions. Persons using documents, however, are 
often interested in a number of otl^r items, including: Making infer- 
ences abo^t. the values, sentiments, intentions, belief or Ideologies of 
the source^ or authors of, the documents ; making inferences about 

group or societal Cor personal) values; and, evaluating the effects of 

'^^<''^. • 

communications on audiences that they reasch (Williamson, et al 1977, 
pp. 291-7.) These foregoing reasons, however, are not a sufficient 
list. An investigator might watit to do documentary analysis because he 
has come into possession of a series of documents which contain valu- 
able information about some inquiry problem of interest, or he has 
sought out such documents as part of an inquiry which he thinks might 
be useful or which might lend greater clarity to his understanding of 
the research setting. As ever, the "great trade-off" rule applies: do 
whaitevef will gain more, better, or less expensive information. 

1 V ■ 

What does it mean to do documenta ry analysis? There are .two 

■ ■ ■ ' I .... — ^ ■ - \ 

separate ways to respond to this question, . depending on 'whether we 
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are talking about the analysis of a single document or the analysis of 
multiple documents. Communication theorists have tended to analyze 
communications in terms of the following questions: "Who says what, to 
whom, how, and with what effect, and why?*' (Holsti, 1969, p. 24). 
The' key words are who (conceptualized as the sender ), why (concept- 
ualized as the . encoding process ),- how (conceptualized as the cjKannel ), 
what (conceptualized as the message , which may address ^^^^^ what or 
how) , with what effect (conceptualized as the receiver's r ejfctlon , or, to 
use a more abstract term, the deco^^g process), and ^ whom (concept- 
ualized as the pecipient ). Thus communication can/ be conceived as 



"composed of six basic elements: a source or lender, an encoding 
process which results in a message , a channeL ;^f transmission, a de- 
tector or recipient of the message, and a ci^doding process " (Holsti, 
19^, p. 24), 



One form of documentary analysis is j^bncerned primarily with the 
"what" or "message" portion of the com^nication . Whether concerned 
with a single document, or with mj^ltipj0 documents relating to the same 



event, or written by the same person^^ analysis which is directed toward 



,the message portion of communicat^n is called "content analysis." For 
instance, a social scientist might be studying the single last suicide 

Ok . - 

note of a deceased person J[or clues to attitudes or beliefs. Or he 
might be concerned with the letters, journals, and last note of a sui- 
cide. Or he might be interested in suicide notes as a class of evidence 
about despondent persons. In all three instances, the investigator 
would most likely use content analysis, at least in part, for his in- 
, quiry-. 
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In other forms of inquiry, the investigator will not be concerned 
with only a single document, bufwith a collection of documents which 
will, in general, display neither the same format, organization or con- 
tent categories, but which deal with different instances of the same or 

- —r^ 

a like phenomenon . That i^ to say, the documents are case studies of 
like events, programs, settings, situations, but they do not all concern 
themselves with the same phenomenon. Several good examples of this 
form of documentary analysis, which is called case study aggregation 
analysis (Lucas, 1974a ;b) would be: evaluation reports from multiple^ 
sites of a locally -adopted bilingual education program; policy statements 
frottn a series of mental health clinics regarding collection of payments 
from indigent clients; or case studies of citizen participation in electoran 
campaigns in two dozen midwestern cities. Each of the\|oregoing sets 
of documents (or case studies, which term is used more broadly here 
than in most social scienceiiterature) contains information which is 
likely common to all settings, but which until recently, was unable to 
1^ aggregated or integrated to provide a body of common understand- 
ings. The case survey aggregation method, developed at the RAND 
Corporation , now allows such diverse , random and often qualitatively 
uneven documents to be aggregated in order to derive new understand- 
ings froih old bodies of literatuf^e) We shall tall^ about both content 
analysis and case survey aggregation method in turn. Before doing so, 

c 

however, some primary questions about the nature of documents them- 
selves need to be entertained. 

Presuming one has come into a body of documents, or has set 
about acquiring them (method of analysis still undetermined)^ how does 
one know that the document what it purports to be? That is, what 

. ■ \ ^ . 
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questions ough^lQ be asked in order to determine that document is 
as represented (whatever other problems with \^e writing that may 
exist because of lapses of memory, bad motives, ^tc.)? Clark's The 
Critical Historian (1967) addressed the issue of "docU|nents genuine and 
spurious" and suggested ^teria by which the credibility of the docu- 
ment might be established as the real document constructed by/ the 
writer . He suggested the following questions (although \ome not 
prove as useful when the sources for documents are fair^^y straight- . 
forward; nevertheless, each has some utility and they bear t\epeatihg>: 

o What is the history of the document? 

o How did it come into my hands? 

o What guarantee is the^e that it is what it pretends to be? 
o lA the document complete, as originally constructed? 
o Has it been tampered with or edited? 

p If the document is genuine, under what circumstances «ind 

for what purposes was it produced? 
o Who was/is the author? 

o What was he trying to accomplish? For whom was the docu-. 
ment intended? 

o What were the maker's- sources of information? Does .the docu- 
ment represent an eyewitness account, q secondhand account, 
a reconstruction of an event long prior to the writing, an 
interpretation? 

o What was or is the maker's bias? 

o To what extent was the writer likely to want to tell the 
' truth? 
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o Do other documents exist which might shed additional light on 
this same story, event, project, program, context? If so, 
are they available, accessible? Who holds them? 

Clearly, some of these questions are directed toward historiographical 
and textual criticism. But a number of them are also particularly 
useful when documents are not ascribed (as many project reports are 
not these days); when one comes into possession . of some form of "copi- 
ed" paper (a photographic reproduction which may have been edited or 
altered); or* when documents are assembled from sources who do not 
wish their identity known (the reader ought immediately to think, of 
Woodward and Bernstein, the secretary at the headquarters of the 
Committee to Re-elect the President and Deep Throat). As a caution to 
the inquirer, when multif)le copits of the sape document are available ^ 
from different sources, one ought ^to^^Kfuire them all, especially if they 
are purported to be "sensitive^" A bit of careful checking, while 
time-consuming , can at least establish , if earlier and later drafts are 
availably, what has been excised from a final dr^ft. The excisions 
themselves may provide important clues^, for further inquiry. Clark's 
warning to historians , while it ought not be make paranoids among * 
educational evaluators and inquirers, bears some mention here: 

"Documents as documents, especially formal documents, - 
sometimes have a senii-hypnotic effect on the minds of 
those who use them, and it is important to remember 
that all documents have been produced by fallible and 
potentially dishonest human beings, and that before 
they reach the scholar they may have .passed through 
' the hands of others who may also have had their fail- 
ings, and were also potentially dishonest" (1967, p. 62). 
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What does the methodology of content*^ analysis entail ? Content 
analysis itself is a term in evolution. Those three definitions which 
seem to come closest to the usage we intend are those of Berelson 
(1952, p. 18), Barcus (1959, p. 8) and Holsti (1967, p. 14). Berel- 
son's definition, "Content analysis is a research technique for the 
o|>Jective, systematic and quantitative description of the manifest con- 
tent of communication , "^relie^ on assumptions of counting, of translating 
the analysis into some form of numerical statement. "The definition of 

Barcus, ^"The term 'content analysis' is used here to mean the scientific 

1 _ ■ * 

analysis of communications messages. . .The method is br^s^ly speaking 
the 'scientific method,' and while being catholic in nature, it requires, 
that ,the analysis be rigorous and systematic , " however , implies no 
such quantification. After some debate on the issue of the numerical 
quality of content analysis, Holsti settled on a definition which makes 
no reference to the quantification " issue: "Content analysis is any 
..technique for making inferences by objectively and systematically iden- 
tifying specified characteristics of messages''' (p. 14). This definition 
^^uits our purposes for the same reasons it suited his: it satisfies the 
three criteria of objectivity, systemization , and theoretical framework 
(although Holsti goes one step further and refuses to limit himself to 
the manifest content of documents, even while admitting that only the 
manifest and not the latent may be coded) . For JLhe purposes of this , 
book, two other observations are in. order. First, in documents (Kinless 
commissioned by the inquirer) the content is generally not^pepifically 




under the inquirer's control. He has to take what he canfgetNand work 
from those materials. Second, as a result of the first, the "specified 
characteristics" of the messages may need to emerge from the material 
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Itself rather than be imposed a priori by theoretical construct. From 
our perspective, this is a most fortuitous circumstance, since it virtu- 
ally guarantees that the categories will be grounded in the data (hence. 
In the context). 

What are the basic characteristics of content analysis ? There seem 
to be fo^r major characteristics of content ylanalyeis on which most 
methodologists agree and a fifth which is in the process of transf6r- 
mation. While different writers may state them slightly differently, the 
basic tenets of what constitutes solid and rigorous analysis are well 
known. First and foremost, it is a rule-guided process . Each step is 
"carried out on the basis of explicitly formulated rules and procedures" 
(Holsti, 1969 p. .3). In order to satisfy the criterion of objectivity, 
and to "minimize. . .the possibility that the findings reflect the analyst's 
subjective predispositions rather than the contents of the documents," 
rules must be derived, procedures clearly delineated, and selection 
criteria -clearly defined. A subsequent, analyst, using the same rules, 
procedures and criteria for selection ought to be able to arrive at the 
same inferences from the documents. 

Second, it is a systematic process . Holsti defined the systematic 
nature of the inquiry as conforming "...to certain general canons of 
-category Construction. .. [so] that the inclusioti and exclusion of content 
or categories is done according to consistently applied rules" (1969, p. 



4). Once tn^ r;^les have been clearly explicated, they are applied in - 
the* same way to all content whether the analyst regards it as relevant 
or not. ; 



Third, it is a process which aims for generality . The" findings 



ought, in the Ipng haul, to display theoretical relevance, or, in the 
case of naturalistic inquiry (to ej^tend the set of characteristics a jDit 
further), they ought to further the development of insights with re- 
spect to context which serve In other instances than the single docu- 

' 2 - ' - 

ment In hand. In commenting about th^ issue of geherality, or theo- 
retical relevance, Holsti drives the point home forcefully (1969, pi. 5): 

"...Purely^ descriptive Information about cdntent, unrelated to 
other attributes of document or to the characteristics of the sender 
or recipient of the message is of little value Such results take on 
meaning when we compared them with other attributes of the 
^ documents, with documents produced by other sources, with char- 
acteristics of the persons who produced the documents, or the 
times in which they lived, or the audience for which they were 
intended. Stated somewhat differently, a datum about commuica- 
tion content As meaningless until it is related to at least one other 
datuin... Thus all content analysis is concerned with comparispn, 
the type of compajrispn being dictated by the investigator's theory." 

Fourth, it deals in manifest content . The investigator is of course 

often interested in drawing inferences from the documents that he is 

examining, but such interpretations are a matter for 'later analysis, 

particularly iti view of the totality of documents (and their type, e.g., 

whether private, or public, formal or informal, official or personal, 

edited versus complete, and the like) to, be examined and other contex- 



tual factors that njight be taken into account. The content analysis 

I 

itself is confined to the mariifest (as opposed to latent) pontei\t of the 
documents, although content analysts themselves (Eferel^qn, 1952 r"^ 
Holsti, 1967) have taken issue/ with the manifest-iatent issue. There is v 



2 

Since the authors have previously attended to * the question of 
generality (generalizability) of naturalistic research, we feel fpee to 
include a somewhat broader definition for the naturalistic inquirer here, ^ 
in' order to make applicable the method under (jiiscussioi^r 
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general agreement that in the. coding stage of research , "... the ^ta^ 
-at which specified words, themes, and the like are located in the text 
and placed into categories, one is limited tp r^ording only those items 
wh^ch actually appear in the document..." (Holsti, 1967, p. 12), while 
at the later interpretative stage, it is generally agreed now to be 
permissible for the investigator to rely upoti his insight, . intuition and 
imaginationxt^ draw inferences about latent content decoding process as 
well as to draw coritlusions about the meaning of the manifest content 

* * * * » 

(encoding process). This broader definition serves the needs of nat- 
uralistic inquirers well, since it is not only semantic symbols which are' 
relevant to the research process, but ."issues" and "concerns" in the 
minds of the project staff, stakeholding audiences, contextual infor- 
mation (including description), and value systems operant in the con- 
text. V 

Fifth, content ^analysis has historically been defined (although that 
definition is changing) as a quantitative technique. Assuming the 
method is strictly a quantitative one allows the researcher a high de- 
gree of precision in the statement of conclusions , ° allows him to assign 
abme degree of confidence to the generalization of results and permits 
certain kinds of numerical manipulations to be performed on the data. 
But arguments ^gainst such a strict quantitative (and toward a more 
qualitative) interpretation of content analyzed data can be made, includ- 
ing: . that frequency of assertion (or symbol) is not necessarily related 
to the importance of that assertion (either to the sender of the recip- 
(lent)/ that occasionally more meaningful inferences may be drWn from 

qualitative or nonquantitativ>e methods , and that often , emphasis on, 

1 • \ ... 

quantification of symbols and.mrecision comes at the cost of problem 

■ . ■ ■ ' ' . ■ • ' ' 
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significance. Holsti cites Tukey, the statistician, on this point (1967, 
p. 12): 

> 

"Far better an approximate answer ,to the right question y 
. which is often vague, than an exact answer to the wrong 
question, which can always be made precise.. Data analy- 
sis must prdgress by approximate answers, at best, since 
its knowledge of what the problem really is will at best be 
approximate." \ 

In any event, both quantitative and qualitative approaches are now 

deemed suitable, depending on the questions which need to be answered 

by the research. - . 

Hoyf that content analysis has been generally described (in methods 
ologiqal terms), spme comment about the actual coding process . seems in 
order. It is not the intent of this paper to explain how one does^ 
content analysis; there are many fine and carefully drawn works on 
that process already available. Nevertheless, t^e first problem beyond 
the acquisition of documents has almost always been what constituted 
good categories for coding. The decisions confronting tl^ analyst fall 
under three general rubrics (Holsti, 1967, 'p. 94): 

"How is the research problem defined in ^rms of categories ? 

What unit of content is to be classified? 

What system of enumeration will be used?" . 
As Holsti notes, these are not separate decisions, but interrelated ones, 
and alway? made on the basis of the original formulation of the reseat-ch 
problem. In short, decisions with regard to tlve^e three crucial ques- 
tions are grounded in the problem to be investigated. " ■ 

"Coding," in its precise definitional sense, "is the {)rocess where- 
by raw data are systematic^ly transformed and aggrfeqated into unitsvl 
which permit precise descrip ton of relevant content characterisUcs" 
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(Holstl, 1967, p,'94). The coding process ought lo be guided by two 
overriding principles: first, whatever method the inquirer may choose 
to code, that method (or set of categories, or unit of analysis) has 
embedded in it assumptions pertaining to, the nature of the data and 
whatever inferences may be drawn from them; and second, tj^pry, 
hypotheses and inquiry questions alone ought to guide , the coding 
process and determine content categories. "In short, unless he can 
state explicitly why he is analyzing documents, .he cannot intelligently 
work out a plan ^n how to do it" (p, 94). The coding process, while 
it is essentiMly an "arty" one, informed by practice, theory, and 
careful reading, ultimately determines whether or not the resultant 
research is worthwhile, for without cpding categories which fulfill 
canons of good taxonomic constructfon , the research is incomplete, 
irrelevant, or possibly in error with respect to its conclusions. 

Canons'-, of «good category.^ or taxonomy construction are five in 
nutnber: 

1. categories must first and foremost "reflect the purposes of the 
research;" the design must include conceptual definitions, which 
are the definitions of the variables with which the investigator is 
concerned, and the operational definitions which specify the indicar 
tors "which determine whether a given content datum'ifalls within" 

a given category; . t ' 

* V ' * 

2. categories must be exhaustive - . that is, each datum must 
finally be able to be placed in one category or another. Often 
this is accomplishe4 by specifying the concepts (or variables) 
which define the study "as precisely as possible by characterizing 
its major properties ; these would serve as rules by which coders 
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would Judge wheth^ content units fall within, its boundaries" 
(Holsti,1967, p. 99). ' * ~ 

3. categories will be mutually exclu^ve — that is, no single content 
datum reasonably fits into rapre than one ceJJ or category; 

4. Categories must be independent— that is, assignment of ^ some 
piece of data will not in any way affect the classificatldti of other 
pieces of data, althoii^s^ this rule is difficult, if noti impossible, to 
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satisfy when "content units are scaled along some dimension," or 
when "some form of ranking is used to assign values to content 
units;" . . 

5. categories must be derived from a single classification princi - 
, pie— that Is, levels of the analysis which are conceptually different 
must be kept separate (see earlier discussion on problems with 
manifest and latent content, and levels of analysis) (derived from 
Holsti, pp. 94ff). 

Given this set of rules , however , it is not clear^ how one goes 
about the creation of categories for unitizing and taxonomizing the 
symbols (or tissues and concerns) identified, and Indeed, there are no 
simple answers to this question, although Uiere'.-are several "tacTes" 
which one may take. The naturalistic inquirer, first and foremost, 

n 

would want some or all of these categories, at least at some point in the 
Inqull^y, to emerge f^m the data. In that way, the classification 
system finally derived will be as well-grounded ^ any. Furthermore, I 
rarely does any investigator adopt the classification scheme of a prede- 
cessor; the emphasis on de novo research, on "unique" or "new^Drob- 
lems, has tended to persuade Inquirers that n^ classifications, iiew 
coding systems were called for. The classical emphasis upon theory- 

■ 30 
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testing, too, as opposed to the creation of grounded theory, has caused 
investigators to assume that new taxonomies were needed. But some 
consideration of the types of categories which have been used would be 
a Useful one, and migl^t serve as a starting point for the evaliiator or 
inquirer who has not ^efore utilized documentary analysis. 

There are two broad .typologies of categories, the "what is said" 
(or subject matter classification) dimension and the "how it is said" (or 
device) dimension., Within those two broad areas, some, though clearly 
not all, of the possibilities might include: 

What is said" categories : 

^ubject matter - what is the communication about? ' |^ 

o direction j how is the subject matter treated (favorably , un- 
favorably, strongly, weakly, humprously, seriously,)? 
o ' values - what values, goals or wants are stated? 
o methods - what methods are utilized to achieve gfoals or 
intentions? -i^ • 

i 

o traits - what *are the characteristics ascribed to the persons 

or contexts described? 
o actors - XQ whom is the performance 9f certain acts ascribed? 

• r 

o authority - at whose, behest dr in whose name are statements 

made? , , : , , 

o origin - where, or from whom, do^s the communication origin- 
ate (the reader will |*emembe)r that often, documents will come 
into his hands , which are, not ascribed; that is, they are list- 
ed as having come from a "project" or "program," or some 
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official source such as an Office of Budget, but they will 
bear no name). 

o target - to whom is the document directed- -to | what person 
or group or office? 

o , locjttion - where does the action take place? About what 
site or event is the document written? ' 

o conflict - what are the sources and levels of conflict? 

o - endings - is there closure? Are conflicts resolved happily, 
ambiguously, or tragically? • . • 

o , time - when does the action take place?^ ]l a series of 

documents exist,^ is there some implied chronological order 
to the sequence of events, or do all documents describe 
different perspectives and perceptions of the same event? ^ • 

« 

"How it is said" categories : 

o form .or type of communication - what is the medium of coimnuni- 
cation (e.g., newspaper, radio, television speech, editorial, 

t 

project memorandum, quarterly progress report, personal letter, 

administrative log or* journal, personal diary, or the like)? 
0 form of statement - what is the grammatical or syntactical form 

of the communication? 
o device - what is the rhetorical, persuasive or propyagandistic 

Method used? (adopted from Holsti, 1967, 99ff) 



Since there are no standard schema or norms ot classification, the con- 
structiQn of categories is often a trial-and-error process, forcing the 
inVestigator^to movesbetween the data and either an a priori or ground- 
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ed theory. "Sof^ts" are performed on items of data; when the sorts do 

' ► ■ * » ' ■ " 

'not account for all units of data, other tentative categories are gener-, 
ated. the process* is completed, using either a combination of theory^^ 
and ddta or a combination of theory ^building or context-construction 

Jpnd data, modifying categories Until ' the . system is complete and each 
datum can be sensibly accounted for. 

The designation of i^nits of analysis, too, is a coding decision 

■ . > 

whish may be decided either prior to or during undertaking the actual 
analysis. The unit of analysis may be either the. single word or sym- 
bdl/ the theme (which is defined as an assertion about some subject, 
e.g.", "Aryan superiority" as asserted by Nazi {propaganda pamphlets 
during World War II), the characters or aptors about whoi^ ths; commuAi- 
cation is concerned, the grammatical unit, pr the type'df item (e.g., 
film, book, newspaper editorial, and the like). Again, as with the ere- 
ation of a classification system, the selection of a unit or units of 
analysis depends upon three considerations: ' (1) which units meet best 
the needs of the researcher, (2) which fit the requirements of resear- 
ch, since the determination of the unit of analysis/ can mediate the. 
rcisUlts of the analysis, and (3) which i^its fulfill the criteria of "better 
data, more data and least costly data" (which garner the most amoUi^t 
of data -^fdr the smallest expenditure of time and resourctes^. 

If the Analysis is to be qarried" out on a quantitative basis (rather 
than purely qualitatively), systems of enumeration must also be devised 
(the system which is devised also formii|g a ^ portion of the body of 
rule's whtch guide the analysis). Systems of enumeration identifiedl thus 
tsr in a, variety of content analyses irjclude measures of time and space; 
Actual appearance of the unit of analysis f frequency of the unit's appear- 
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ed theory. "Sorts" are performed on items of data; when the sorts do 
not account for all units of data, other tentative categories are gener- 
ated. The process is completed, using either a combination of theory 
and data or a combination of theory -building or conte>|t-construction 
and data, modifying categories until the system is complete and each 
datum can be sensibly accounted for. 

The de^gnation of units of analysis, too, is a coding decision 
which may be decide^ either prior to or during undertakfeig the actual 
analysis. The unit of analysis may be either the single word or sym- 
bol, the theme (which is defined as an assertion about some subject, 
e.g., "Aryan superiority" as asserted by Nazi propaganda pan4)hlets 
during World War 11), the characters or actors aboi^t whom the, communi- 
cation is concerned, the grammatical unit, pr the type of item (e.g., 
film, book, newspaper editoriaK and the like). Again, as with the cre- 
ationTof'a classification system the selection of a unit or units of 
analysis depends upon three considerations: (1) which units meet best 
the needs of the researcher, (2) which* fit the requirements of resear- 
ch>s^since the determination of the unit of analysis can mediate the 
results of the analysis, and (3) which units fulfill the criteria of "better 
data, mbre data and least costly data" (which garner the most an^oiint 

If the analysis is to be carried out on a quantitative, basis ^(rather 
than purely qualitatively), systems of enumeration must also^be devised 
(the system which is devised also 'forming apportion of the body of 
rule's which guide the analysis). Systems of enumeration identified thus 
felt* in a variety oiNcontent analyses include measures of time and space ; 
actual appearance Vf the unit of analysis; frequency.. of the unit's .appear- 



of data for the smallest expenditure of time and resources). 
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•ance; and intensity (of particular use when the analysis involves ^ti- 
tudes, beliefs and values), which is often measured on a scale. 

As with any method qua method. in the sociail sciences, t4|^ deter- 
f mination of how one might sample (if the universe of documents [jiertain- 
ing to the phenomenon under investigation is available, and some 
choices must be made as to which will be analyzed), how reliability (or 
relevance), and validity (or fittingness) might be established, are 
unique to some, extent to the method. The inquirer seeking to carry 
out a first content analysis ought to make himself familar with classical 
methodological works in content analysis, and adapt the method to his 
particular needs. Ai>^important caution to remember is that this meth- 
od, like the next one which will be discussed, is a rule-guided proced- 
jire . When rules are clearly specified, and categories clearly defined', 
an independent researcher ought to be able' to arrive at the same re- 
suits using the same documents for analytic purposes. That is to say, 
the research process itself- -whether or not independent researchers < 
might agree about the taxonomy— ought to be duplicable if the same 
rules and procedures are followed. . 

1 

Is documentary analysis like any other form of analysis , the in- 
quirer might wonder? Indeed it is. Bqgdan and Taylor (1975, pp. 
95ff.) in their work on qualitative methods, place both personal docu- 
ments and. unstructured interviews into the same chapter, primarily 
because they can be dealt with in parallel ways. We would suggest^ that 
the basic problem of formulating (content analyses is identical with those 

of ormnizing and interpreting either, notes from unst^-uctured interviews 

* 3 
or from participant observations.. 
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• i In fact, lists of behavior types,, or participation/non-participa- 
tion observation "schedules" or protocols represent nothing less than 
a priori - categorization , vinitization , ^pr^ taxonomic representations of what 
the observer might see. 



Everything we have said under those headings has relevance here, and 
everything we have said in the foregoing discussion also has relevance 
In those other two inquiry situations. The creation of classification- 
systems, the decision -making with regard to uni^ of analysis, and tl>e 
formulation of taxonomic headings for subjects, concerns, issuer or 
behaviors under investiga tlon — all utilize the^ethods of content anql- 
ysis, and abide by the same procedural canons. The methodologies qua , 
method in all three areas are virtually identical. 

What does the methodology of case survey aggregation analysis entail ? 
As proposed by William Lucas (l!&74a;b) the case survey aggregation 
method is a means for aggregating "... diverse case studies together 
under a common conceptual framework so that findings will be cumula- 

\tive...to identify what it is we already 'know,' and what, it is we do not 
know, and what it is we suspect" (1974b, p. 1) The term case study , 
it should be made clear, is used in a somewhat broader definitional 
sense in this instance than is normally used in the social sciences . 
Case study in this particular method means any descriptive or ev^u- 
ative analysis of a common social unit, or a local program, or of some 
agency. Fpr instance, included in, the "case study" category might be 
"clinical studies of individuals, administrative studies of organization, 
anthrc^ological reports on primitive societies" (1974 a, p. 8), evaluation 

^reports, and activities reports on local programs (1974b, p. 1). The 
strength of the method lies "in its capadty to integrate the findings of 
diverse studies about organizations and programs. It is more /flexible 

. in that many different types of studies using different measurement 
tectoiques can be brought together, and new concepts can be devel- 



oped and considered that none of the original research ever addressed" 
(1974a, p. 12). Thus, the documentary case studies utilized for this 
type of research may be dramatically dissimilar in form or coverage from 
one another, so long as they are descriptive of some common social unit 
(e.g., a series of mainstreaming inservice projects at each of fifteen 
xUfferent locations in the United States, the clinical studies of six 
paranoid schizophrenics, each of. wh|)m believes he is Napoleon Bona- 
parte, the evaluations of a dozen sites where NSF curricula are being 
adopted, and the like). 

What are the basic characteristics of the method? The essentials 

7 ■■ ■ ■ — ■ 

of the method are six, which will be reviewed briefly here, then des- 
cribed, utilizing an example with which we are familiar. Required-^. are: 

1. a cliecklist— defined as a set of tightly defined questions and 
answers intended to ascertain information " about certain out- 
comes of interest ^and t^e alternative determinants of those 
outcomes. Both the questions and possible answers to the, 
questions emerge from the research problem and/or from a 
guiding theoretical paradigm or model. 

2. a set of rules, which guides the search for, and the sampling 
of, case studies. Since the case survey method is based 
upon the "universe" of all such case studies (which pertain to 
the common social unit under investigation), the search and 
sample rules are actually inclusion/exclusion rules which are 

formulated beforehand so that bias in thi^ selection (assuming 

■ ■ }. 

availability of the universe of case studies) does not occur. 
• 3. a set of decision rules to be followed in dealing with the 
questions; 
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4. a group of reader-analysts who will apply the checklist to all 
of the case studies; 

5. a confidence scale on Which each reader-analyst may record 
his subjective impression about the level of confidence he has 
in any judgment which he makes ; and 

6* a means for checking reader-analyst reliability. 

Th? Checklist Some illustrations with an example may prove fruit- 
ful here. The study (Lincoln, 1977) was undertaken to explain why 
certain predictions made by competent sociologists of education (Sieber 
and Lazarsfeld, 1964; 1966) in the mid-sixties about the important 
future of bureaus and institutes of . research, development and field 
service in schools of education did not come to pass . Sieber and Laz- 
arsfeld had studied major "bureaus" around the country, and had 
arrived at certain propositions and conclusions about their potential in 
the research and development %rena which, if sustained, would indeed 
have meant a bright and productive future fpr such units. It was felt, 
however that some, if not all, of the assertions about the future of 

these units had failed to materialize for a variety of reasons, e.g., 

I 

Congressional support for certain kinds of researcher training had not 
been forthcoming, or that certain assumptions about the nature of 
organizations made by Sieber and Lazarsfeld turned out to be invalid as 
demonstrated by later, research in organizational theory and behavior, 
or that fiscal crises precipitated other forms of organizational action to 

/ 

{ 

support research and development activity. For many of the Sieber and 
Lazarsfeld propositions, it was possible to structure certain alternative 
hypotheses . or "counter-propositions," based on recent documented 
history on research developments or discoveries. A sample set of. 
propositions, with, their counter-propositions looked like this: 
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Proposition . Counter- Proposition 

Research is a faculty priority, . Teaching, not research, is a 

and when allowed to choose, fac- faculty priority is all institu- 

ulty members overwhelmingly will tutlons except private, doctor- 

choose^esearch over teaching. al level rej?earch-oriented in- 
stitutions. 

Team, /especially interdisciplin- Teaml-type research 'is not neces- 

ary , research is better than re- sarilyl better than research done 

search done by single individu- by thjp lone researcher. 

■ \ ... I 

The reward systems of universi- Reward systems of universities 

ties are, or will be, accomo- are, by and large, not multiply 

dating of multiple modes of re- focussed but singly focussed, 

search (e.g., teamwork modes). rewarding the lone researcher. 

University-based research organ- University-based research organi- 
izations grow out of the needs of zations — like most research sub-r 
researchers to increase their own units--grow out of administrative 
opportunities for serious schol- desire to create a unit to pro- 
arship. tect ^'precarious values" ^from 

erosion or attack". \ 

The list of questions which reader-analysts were to respond to 
grew out of these prepositional areas, which feir into five conceptual 
categories, to wit: goals and missions of the bureau, group processes 
and faculty reward systems, activities and work roles of faculty, unit 
integration (degree to which the bureau was integrated with the school 
of education at large), and individual perceptions and motivations of the 
missions and processes. The list of propositions and counter-pr'oposi- 
tions gave rise to both the questions and the alternative determinants 
of outcome, listed in the form of possible answers to the questions. 
For example, given the last proposition mentioned above , . the checklist 
contained ^ this item (among several addressed to the Same conceptual 
category): . , 

What apparent motive was employed in justifying the creation "bf 
such a un|^t (i.e.,, a bur^u, institute or center of educational 
research, development and/or\ service)? 
\. - 
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1. To create opportunities for extended or interdisciplinary 
scholarship. 

2. To sei*ve as a "holding company" for logistical or fiscal 
purposes. , : 

3. To stimulate and model behavior for extra-bureau facul- 
ty. ' A 

4. To serve as a research resource and advisory facility. 

5. Other (please describe) . 

6.. Cannot tell from the case study. 

Sixty-four questions and determinants of outcomes were thus generated 
to probe the five conceptual areas. Each of the question and answer 
sets was applied to each case study by each of three reader-analysts 
(since* the n of the case studies was small; if the volume of case^ studies 
in the universe were large, each reader -analyst would only do a p^t?ti^ 
of the total analysis)'. - \^ ° , 

Inclusion/exclusion i:ules (searching^ and sampling procedures) for 
case studies . The first step in the process is the genera tioiy of a tight 
definition of the phenomenon under investigation. In this instance, 
only education research, development - and service units housed in 

/ * * 

schools or colleges of education were considered. The possible universe 

• A 

of such units was determined by a search of RITE proj^t (Clark and 
Guba, 1977) files and a survey of deans of such unit§ mroughout tjie 
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RITE is an acronym for "Research on Institutions of Teacher Educa- 
tion," the name for the national study of schools, colleges and depart- 
ments of education- conducted by Clark and Cuba. 



country," which had been undertaken to solicit informatibn about the! 
existence of such units."* All such unit directors (pr their deans) were 
solicited ior documents pertaining to the bijreaus— histories of the 
units, annual reports, staffing policies,' fiye-year plans, budgets, if 
available, and the like^ All documents provided were included/ except 
that highly redundant documents were excluded and documents which 
contained information on fewer than three outcomes of interest, (of five 
-brb^d categories) were discarded on technical adequacy grounds. * 

Decision rules for questions . Concepts relevant to the inquiry, 

e 

r 

e.g., reward systems for bureau members, budgetary arrangements for 
support of the unit, unit missions, and the like were defined. Tlu:\ee 
pre-analyzed case studies were provided to reader-^n^lysts for guidance 
on how definitions were to be applied. 

( 

The reader-analysts . Three reader analysts 4irere selected who 
had credentials both In the area of educational research and develop- 
ment ai>d in organizational theory . One had had actual experience 
directing a ^^bureau of research and development. Each was trained In 
the definltibns and concepts, "walked through" one case study with the 
investigator, then allowed to complete three case studies (discarded for 
historical bias from the final sample) In order to establish reader relia- 
bility and inter-reader reliability coefficients. 

Confidence scales . Each Item of the checklist was accompanied by 
three measures of confidence. The first was the option to signify that 
t^e. case study did not contain any information relating to the Item of 
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Interest. Second, the analyst was a^ked to provide, if he or she 
could, an actual page number Iri a sp«:lfic document that supported the 
answer given oT judgment made. Third, for each item the analyst 
Indicated his confidence in the judgment on a five-point scale ranging 
from "1"— high confidence— to "5"— no confidence. 

- Inter-reader reliability measures . Reader-analysts were personally 
and Individually trained by the investigator using pre-analyzed cases. 
In addition, each reader-analyst was asked to* complete three "trial" 
analyses (utilizing the cases not utilized in the study) 'which could then 
be intercorrelated. Reader-analysts who might not have shown a suffici- 
ently high correspondence with ratings of peers and ^ the investigator 
would have been eliminated (although, in fact inter-reader reliability 
estimates were all .75 or better, both' on trial cases and on the actual 
case stijdies). * . 

Procedures . As is the case with content analysis, the generation 
of categories of interest is dictated by the questions of research lifter- 
ei^. LUcewise,' §ach step of t^e inquiry process is guided" by rules 
which are developed in order to eUmi^ate bias in the sampling and 
searching to establish reliability and^ validity, and to insure that. rules 

are^sufficiently clear and definitions sufficiently rigorous that the study 

>■ ■ " " .. ■ 

could be repeated by anothe^ set of analysts using the same rules and 
procedures. T ' 

Once all case studi^ have been analyzed with the checklist, re- 
sulja can be tabulated either by^ndv(lf' the . number of case studies is 
relatively small) or by computer.. In the particular case cited, all case 
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studies were analyzed by each of the three reader-analysts, although 
this might not always be done. Because inter-reader reliabilities were 
not perfect, occasionally the investigator had to "adjudicate" responses 
herself— that is, return to the case study and determine the best poss- 
ible answer. Frequencies were tabulated and conclusions drawn from 

♦ 

the 64 items under the five broad organizing rubrics. 

While the foregoing discussion should not be taken as a complete 
'expUcation of the case survey aggregation method, nevertheless, it 
jihould be sufficient to demonstrate that meta-analyses of data— especial- 
ly of data which were intended for other purposes— particularly policy 
analysis, is an extremely useful technique with certain kinds of docu- 
ments. 

Documentary analysis cai> therefore proceed frota two different 

* 

directions: either analysis of the documents^ and their encoded and 
decoded messages themselves, and as representatives of broader classes 
of theoretical rubrics for which they were not origi^allf intended but 
for which, when siUbjected to certain aggregation techniques, they may 
yield certain new and broacier forms of inquiry, data. Analysis of 
records, the second technique for dealing with written accounts which 
are official, proceeds along a sligAt different track. ' . 

Analysis of records . 

A rebord,. as noted before, is a written residue attesting to an 
evient or transaction, or providing an accounting. The types and 
numbers of, records which exist is probably unknown and unknowable, 
and the number grows, due to computer capabilities for storag[e and 
retrieval/' daily, The kinds of records which exist can be categorized 
In several ways, for instance: 1) who holds them— Federal^ state. 
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local, country or municipal authorities, private corporations, credit 
corporations, health organizations (including insurance companies and 
hospitals), or banks, to name a few; 2) to whom they are legally access- 
ible—those accessible to the public, those inaccessible to the public, or 
it least to the public at large, although they may be accessible to 
individuals (for instance, credit bureau ratings), and those which are 
hot mentioned specifically by law. The types of records which are, for 
all intents and purposes, public, vary from state to state, although the 
amount of Information available to reporters and the general public is 
increasing rapidly as a result of campaign reforms , , and the push for 
political disclosure (Anderson and Benjaminson, 1976). 

The first and greatest injunction to anyone looking for official 
records is to presume that if an event happened, some record of it also 
'exists. Once the event is presumed to have happened, then the searcl) 
> for the record can begin. But how does one determine what records he 
needs to look at in any given situation? The metaphors of investigative 

reporting and good detective work provide the answers. How does the 

X 

ii^vestiga^r find the trail he wants in the apparently untrackable wilder- 
ness of records he might consult? Williams (1978) suggests a form of 
reconstructed logic for the investigative reporter which he states th\\s : 

"The veteran hais ingrained within himself a special style of reason- 
ing. He knows how things normally work. If he observes a 
phenomenon, an effect', he wonders what caused it. He develops a 
hypothesis and begins checking it against observable facts. He 
works to back up the chain of facts, searching^ for information that 
, will either support or negate his hypothesis.' He tries different 
combinations of. conflicttijg versions of a story until he finds the 
one in which salient points overlap" (p. .13).. 

In a similar vein, Locklin (1976, p. 7) comments: 

"I have a working theory that/ if I know something , if I know what 
the situation is, the date the/ money went, Jiow much money went, 
who paid, who got, if I know that, t can usually prove it." ^ 
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The central notion here is tracking . Guba (1978) has identified "a 
cc»nmon approach to the 'original research' which seems to be based on 
tracking . Actions of persons, whether legal or illegal, inevitably leave 
tracks : If one knows how things work, and if pne suspects ^lat a 
certain action has occurr>ftd, one can imagine what track it must be leav ^ 
"tog; one then looks for the tracks , which has heen 'warping and weav- * 
ing' with the other circumstances of the matter, and one usually finds 
.them, if they exist at all. Absence of tracks cannot ^be taken as an 
indication of innocence (the tracker may, after all, be inept), but their 
presence is proof positive— they constitute the 'smoking gun' " (p. 48). 

While the image of a 'smoking gun' may be a bit strong for the 
educational evaluator, there are many parallels between th6 investigative 
journalist's use of tracking and the reconstruction of events and causal 
connections for the evaluator. A less likely parallel betwegn the two^ 
professions is the likelihood that the educational evakiator in^ht en- 
counter dishonesty, graft, corruption, or any of the social ills that the 
investigative reporter makes his daily fare. More often than not, the 
evaluator will practice his craft in and around projects, programs and 
schools where those who operate them are men and women of consci- 
ence, honest and simply trying to get a job done. The sites will thore 
often than not be extremely public settings (e.g., public schools or 
universities), funds will likely be public (state or federal funds), and 
records which need to be perused will likely as not be public, since 
disclosure of records and sunshine laws make such records accessible to 
anyone . who needs or wants them, or at least to those who - have a 

legitimate right to them. For the educaticJnal evaluator who finds that a 

■ • ft ■ 

project director (or other €\flministrator) decides to be obstructionist 
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and withold documents and records, there is usually someone else on 
the project who will give t^iem up. 

Hqw do I handle the records that I get ? With respect to the 
notion of tracking, Guba (1978) "comments that for the Investigative 
journalist, "The reconstruction of tracks \ad their verification is per- 
haps the most distinguishing and "Characteristic feature... It is by that 
process that he keeps his task manageab^, provides direction to his 
activity, and ki^ows when he has reached a point at which he can 
comfortably stop looking further" (p. 49). The educational, evaluator , 
however, unlike the investigative journalist, would rarely, if ever, 
operate on \ clue or 'tip* that something illegal were taking place. 
Rather he would generate a series of working hypotheses (grounded in 
the context itself) about what was happening on site. Starting from 
those hypotheses, he would see|c the records (and documents) which 
allpwed him to either confirm or \disconfirm his hypotheses. 

Great similarity between the met^j^s of investigative journalism 
with respect to records ana Scriven*s, jiiodus operandi or MO method 
CV974) with respect to evaluation exists, and it is useful to look at the 
verification pf cause-and-effect relationships using latter. The MO 
strategy is one that: 

1? establishes all probable causes for an event or situation. 

■ * . »■ 

V 

2. checks to see which probable causes are in evidences. 

3. ' checks fpr the causal chain, which is characteristic of each of 

the probable ca^ises which are present, 

4. labels as tiife most likely cause that one whose characteristic i 
causal chain is completed. 



The tracking strategy of investi^tive journalism- follows a similar path . 
Most writers in the area appear to imply that one must develop a work- 
hypothesis with respect to the- subject of interest, detern^ine which 
♦tracks* or record trails would be left if the working hypothesis were 
true, and then check the appropriate records to confirm or disconfirm 
the hypothesis. For the educaltional evaiuator, the notion of tracking 
falls^omeWhere between what Scriven suggests that the modus operandi 
methpdologist perform and what the investigative journalist does. Wlvlle 
the Journalist clearly is looking for cause-effect relationships, the 
evaiuator is searching for evidences of relatedness <Guba, 1978).. 
There are mai^y methods other th^n records analysis which test for 
cause-effect relationships in much more controlled fashion; and control, 
particularly of environments and contexts, is not what the naturalistic 
evaiuator is seeking. The journalist (or^ detective) is probably also 
cojtistrained to search for pre-existent records the validity of which 
(because they are generally public and legal documents) is at least 
partially established. For thte naturalistic evaiuator, the records may 
^^ean not only what is already written, but what can be "generated by 
additional procedures undertaken by/ the educational evaiuator (a mode 
.*of operation which... is not available to the! investigative reporter in the 
large majority of cases)" (Guba, X978, p. 108). 

Guba cites three cases which are excellent exemplars of the work 
of the investigative reporter, the. forensic pathologist (on whose work 
.the modus operandi method is built), and the educational evaiuator 
which bear repeating here. A typical problem for the investigative 
reporter might run thus: 
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A municipal officer accepts a bribe to sway the administration on a 
certain issue favorably to the officers "client." The client cannot 
p«Y the bribe directly in case since it is likely that the IRS would 
be ^i^tracted if the officer were suddenly to deposit a large, unac- 
counted<;for-8um. Instead, the client sells a desirable piece of 
property*" to the officer at Ijargain basement rates; he in turn 
resells the property at its true value. The officer of course pays 
the tax on the capital gain, but the remaining money is 'qlean' 
(having been 'laundered') and can be deposited without hesitation. 
But a trail has been left in the transfer of deeds, yvhich the 
shrewd investigative reporter may uncover by an assiduous search 
6f courthouse records. Of course, in real life this story is likely 
•to have been complicated by the intervention of a dummy corpor- 
ation to whom the client sold the property and which then resold it 
to the municipal officer. But those transactions also leave re- 
cords; the tracking task i^s more difficult but the principle is the 
same. (1978, p. 106) 

The forensic pathologist operates in much the same^way, since "opera- 
tive causes 'fulfill their MO contracts,' i.e., leave the full characteristic 

causal chain behind" (Cuba, 1978, p. 107). It Is this causal chain for 

: ■ ■ r 

which the pathologist also searches. The example ^ given Is one which 

seems to routinely plague a favorite TV sJhow: 

for example, a ijofpSje^were discovered which has both traces of 
poison and a gunshot wound in evidence, the killing cause is the 
one whose full causal chain can be traced. If, for example, the 
poison were of the type in which oxygen/€ffonfe-4n the blood were 
replaced by atoms of the poison, thereby effectively shutting off 
respiration, a chemical test of blood samples would show whether 
any appreciable number of red blood cells were affected; if not, 
one may conclude that it was the gunshot woUnd which did the 
trick. Conversely, if there were little or no bleeding around the 
gunshot wound, one could probably conclude that the effective 
agent was the poison. Checking on both characteristic causal 
chains would .establish almost certainly whether gunshot or poison 
was the fatal agent (1978, p. 107). ^ || 

For an educational evaluator, 'application of the process would 

likely be slightly different, although the same principles apply. Sev- 

5 

eral examples cited' ^y Guba in the evaluation arena Incttide the follow- 
ing ones: ' ' . ^ • 

If It 10 asserted that a pupil's motivation to read is raised by a 
certain teaching approach, that assertion can be checked by look- 
ing at existing library withdrawal records, increases in sales for 
books of a certain kind at local bookstores, etc... [Or] If it is 
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asserted that *a certain instructional outcome occurred because of 
the serendipitous introduction of certain originally unplanned 
teaching techniques, the existence of those techniques and their 
day-to-day impact on the classroom can be at least partially assess^ 
• ed by consulting log descriptions of classroom interactions. .. {Or] 
If it Is asserted* that evaluation reports, have systematically 'turned 
of^' certain -parent groups in the community, some insight might be 
gained Into tiits^iSroblem by re-reac^ing local newspaper accounts of 
PTA meetings at which these repprts" were disseminated (1978, p. 
* 108), 

Clearly, there are an infinite ^number of ways in which the trac](fing 
technique can.be used.. The practitioner gf naturalistic evaluation will 
have, in retrospect, any number .of recollections of how this might have 
•been applied in his own work, and will be able to generate a number of 
instance? where it will be useful in the future. . . 



Are there other techniques which are ]^useful in records analysis ? 

- The answer to that question Is most assuredly yes. The inquiry 
techniques which remain to the evaluator are three in nymber: tri- 



3 



angulation, "circling and shuffling," and "filling," the latter two^^a^ 
which are journalists' terms for the certain processes of iteration, that 
Is, returning to source? to verify and fill in gaps in information. The. 
analytic technique whlclT ^ill b e discussed shortly 4s really a method for 
making sense of records, for being iCle to. retrieve -them and utilize 
them to complete pictures. It reiates to the creation of indices and 
summaries which show interrelatedness between actors and contex^is. . ' 

Triangulation . Triangulation is not new in social research. It is, 
4n fact, an old concept both to sociology and to anthropology, where it 
kas cdme to mean the process of "comparing and contrasting information 
drawn from difference soui;ce8, ''and/or determined by different method- 
ologies" (Guba^.:1^78, p. 116). Triangulation is useful for verifying 
information on the same event from different actors or participants, and 



also for allowing more confidence In data generated by different method- 
ologies. Webb, et al. (1966, p. 3) contend that: 

"Once a proposition has been confirmed by two or more measure- 
ment processes, tbe uncertainty of its interpretation is greatly 
reduced. The most persuasive evidence comes through a triangu- 
t lation of measurement processes. If a proposition can survive the 
onslaught of a series of Imperfect measures, with all their relevant ' 
error, conflde]^^ should be placed in it." 

But trlangulation is important , f rpm another perspective for the 
naturalistic inquirer. Presumably one of the most important strengths 
of naturalistic inquiry is its ability to divorce Itself from the unidiraen- 
sional, value-consensual paradigm which has guided social action re- 
search and evaluation. The process of trlangulation permits multiple 
value perspectives to emerge from the- same context or event, and al- 
lows .for their explication and presentation alongside each<other. In the 
course of checking . "facts , " the naturalistic inquirer causes dif^^ring 
perceptions andv^values to surface. That multiplicity of values then 
becomes warp and weft of the contextual fabric... - ' ^ 

Circling and shuffling . Each of ute^^ terms is borrowed from the 
. Jargon of investigatiye journalism. -They signify the steps taken to 
extend, complete and^ bound an inquiry. "Circling" as defined by 
Williams (1978, p. 80) -is th^^s^rocess of. tciking data or information 
collected from a single, sgurce and "running it back around your circle 
of contacts for refutation or confirmation'." For the purposes of jour- 
nalism, this particular tactic Involves talking with those who are essen- 
tlally presumed to be essentially cooperative. "Shuffling" proceeds from 
the assiunption that' the contacts are non-caoperatlye, that is » • they are 
l^DStile, may be in league with the subject of^e investigation; and the 
qjuesticping will likely proceed from a confllctual, rather than a cooper- 
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ative, mode. The purpose of shuffling is not simply verification or 
discoiiufirpaatlon, however. It. is to* some extent extension of information. 
Not only! will information from one source be checked against the stories 
of others! (who are hostile sources), but some new information is sought*. 

FiUiim . "Filling" performs two functions: it both sets the bound- 
aries of tne Inquiry (which are also being set by other means, e.g.,. 
time", moneV, resources, unavailable records, and the . llkej), and com- 
pletes the picture within those boundaries. The process of setting the 
boundaries,! on .the one hand, and filling in the spaces of the bounda- 
rles, on tMe other, is essentially that of achieving convergence and 
divergence in naturalistic inquiry,. 

" Those concepts appear to have utility for the naturalistic eypiluator 
as they do for the Investigative reportelr or homicide detective. And 
indeed, they need not be reserved for records analysis, but can be 
used <and ought to be used) with all forms of data collection which the 
naturalistic inquirer chooses .to use. That Is, the techniques described 
above are not in i any way limited to simply the skills of this paper. 
They can aijd probably ought to be extended to each and all of the 
methodological techniques In the armament of the naturalistic evalua tor. 

* •> . ' 

How does one Integrate and utilize tJie information collected from records ? 

* . 

Unless one Is a historian searching for a single document or record, 

« . ^ 

.probably no single record will suit the purposes of the overall inquiry. 
Once again, the teclyiiques of the patient detective or the investigative 
Journalist prove useful. Files must be created which essentially cross- 
index, information and actors, and it is from these careful summaries 
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and indices that "a coherent, picture" begins to emerge. The steps are 
basically these seven: 

o A filing system is initiated bv creating a folder for each 

person known to be Involved in the inquiry . .One. may begin, 
in fact, with a single folder for a single person known to the 
investigator. Into this file ought first to go ^ detailed des- 
cription of everything which is known about the perton (a 
personal and/or professional history), and copies "of what 
6fhiBr8 have written or reported about the person*, 
o As other persons become involved , or a^ the principals enter 
into transactions (fe.g:. hold meeting*, have lunch, transfer . 
' fttoperty between themselves^ and the like) with each other , 
^ transaction folders are developed . Each "deal" or transaction 

has a folder of its own, and each transaction is cross-indexed 
f with the original . personnel folder (s). 
o Profiles are developed on each of the actors . Information rer 
latlng to each Individual's background and operating style are 
. added to his folder during the inquiry as the original material 
is fleshed out and information is received, 
o Chronologies are developed for the varfous transactions . When 
information becomes avaUable (or is located), at first, it may 
not appear to fit a pattern. But by keeping dated chronol- 
ogies « events and transactions become more easily traced both 
backward through time and predicted for the fuj:ure. As a 
result we may find that on January 5, Mr, Smith had lunch 
with Mr. Jones; on January 5, Mr. Jones withdrew $5,000 
from his back account; oh January 6, Mr. Smith deposited 
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$2,500 in hi|S account; on January 8, Mr. Jones was awarded 
an important contract over which Mr. Smith's office h€^d 
jurisdiction; and on January -07— J4r. Jones deposited an addi- 
tional $2,500 in his account. 

Important items and events are cross-referenced as the fijes 
are developed . Thus, the transaction given as an ' exam^ple 
above might ultimately be note4 both in the files of the per- 
sbnnel and In their transactions files. Notation indicating 
cross-referencing would also be included in each file. 
Files must be summarized on a systematic basig . In this way, 
the evaluator or investigator need not deal with original 
material each time he returns to the file, but rather has 
available to him a summary of what the file contains with 
which he may work. 

Entries in the files will give rise to hyp^eses which must 

J)e checfegjl or which point to information gaps which need 

to be closed . It is quite possible at tltisLDoint thaj/^e 

original hypothesis which iniU^ted'^theinquiry has blossomeU 

into several hypotheses (di4 Mr. Jones bribe Mr. Smith?), or 

that the inquiry may have become extended in such a way as 

c 

to include a web of events and transactions quite larger than 
the investigator originally expected. The reader is reminded 
that the original Woodward and Bernsl^ein inquiry began as s^n 
interest in why a number of "third-rate burglars" would be 
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Interested in the files of a weak presidential candidate . 
Before the inquiry ended, it had toppled the "imperial Presi- 
dency" of Richard M. Nixon. No suofi effort was ever origin- 
ally intended; but the burglary, of course, led to a web of 
r complex interactions and relationships which have not yet 

been fully sorted or clarified. 

This process of developing and keeping files, cross-ljjdexing materials 
und references and building chronologies appears to have many appli- 
i:a|ions for the naturalistic inquirer. The utility of such a process, 
esj|)ecially for larger inquiries and evaluations , where the simple manage- 
in|fent of an enormous flow of data presents problems, is clear. , 

Can records be trusted to be accurate ? y/ 

The answer to that question is probably most of the time, yes, but 
sometimes, no. 'to trust records entirely is to be naive and to ovejrlook 
the possibility of simple human error, forgetful clerks, or deliberate 
lying oA X^e part of those who furnish infonniation (i.e., misstatements 
a::oncerning taxable income, the value of property, the number of de- 
pendents, and the like). There are at least six factors which bear 
upon, the accuracy of records ■ which one might want to use. They 
include: / ' , 

1. errors introduced by persons to whom the reco^i^ pertain, 
either inadvertently (not being quite sure, for instandeif the 
value of the real estate one owns), or by intent (as in the 
examples above) • 

■ ■ , ■ . • ■ r.\ . . ■ 
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errors introduced by data collectors who devise the records 
(e.g. failure to count all the legitimate residents in a city 
block, misinterpretation of information received,, transposition 
of numbers onto permanent records, etc.) 

errors introduced by recording or filing mistakes. Clerical 
, mistakes and filing errors make some records either incorrect 
or totally inaccessible . 

changes in record-keeping systems which make some records 
non-comparable to others. If a local police department chan- 
ges th^ way -it defines "breaking and entering," for example, 

' the , statistics on '!the rate of that crime may change even 

i 

though the true ^ rate remains the same. In the same way, 
changes in the ' 'frequency or thoroughness with which some 
events are recorded may alter "true" statistics. One contem- 
porary example would be the reporting of rape. It is not 
clear whether the incidence of rape across the U.S. is on the 
uprise, or whether the new feminist consciousness, coupled 
with more sensitive police handling of such crimes and their 
victims, has tended to make the reporting of them more 
common^ » * 

err9rs due to historicity or. temporal chakgett.. . Information 
which has been recoPdiid at some ^?|^^t in^^^^;tme .may .have 

has not been undated periodically. 



become obsolejte if it 

Changes in the number of children encompassed by a family 
provide a good example. The birth of another child, if not' 
recorded in the file, could alter tax records and the like.' 
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6. factx)rs which enter into the development of "official statis- 
tics." .The reader Is referred to the excellent dissertation by 
. Johnson (1973) on the "The Social Construction of Official 
Information." Johnson spent a year as a participant observor 
^ ' ,i ^ social welfare agency, and documents in detail how 
"records" are generated In such agencies. In particular the 
required statistical reports which become the "official infor- 
• matlbn" for the agency. To quote Johnson (p. x): 

"To successfully complete sudh statistical reportlngs , In 
every Instance the social workers were asked to 'reduce' 
their knowledgeable understandings of their activities in 
the linguistic terms provided by a given report. As) the 
members were held rationally accountable to so many differ- 
ent administrative structures and In so many different 
terms. It was only by making use of one's understanding 
of the situated official reporting context which allowed for i 
the possibility of 'making sense' of a given report, and to 
further impute the Intentions of the reporting format. 
This means that . the situational reporting context is not 
only partially independent of the organized features of the 
.other official work contexts, but furthermore, this was 
coAimonly known <by the social workers as one of their 
^ organizational facts-of-life. By making use of such an 

j understanding, the . sociaK workers and/or administrative 

^ * personnel in these welfare agencies used -official 4p<^^6i^^ 

" and statistical reports in a continually phgoing'mid self- 
organizing attempt to change or stabilise tjfe nature of 
their everyday practices in accord with their individual 
motives, intentions, plans, dreams; fears, hopes and so* 
i ' on." • - 

. Xp translate, Johnson Is saying that even official reports, are re-* 

, sponded to In terms of who gets the report, .what its perceived 

* ' . i ' 

^ * purpose is, and how the respondent perceives It will have an 

♦ ■ ■* • . 

Impact on him as a person and worker. Official data are. In 

■ ^ . . \ - ■ ■■ ■ • ■ 

short, doctored by respondents from a variety of perspectives. 
For those reasons, and no doubt many others. It would be un- 
. / sophisticated In the extreme to accept documents and records at face 

^ ■ value- The^wary inquirer wiH attempt a variety of means to certify or 
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warrant that a record is accurate as represented, including checking 
other records whiph might triangulate, the first and simply asking (at 
some poin\ in the inquiry) the person about \vhom the document ob- 
tains. 



Are there special ethical problems in the use of records ? 

The answer is of course, yes. This is especially true of records 
which may be personal, private, prohibited, or simply not well covered 
in terms of access jprinclples. State law (which varies from state to 
state) and the federal Freedom of Information Act make clear which 
documents will be available to private citizens and which will not. 
Among those prohibited records from the latter regulations would be 
such records as pertain to classified defense or foreign policy informa- 
tion, personnel rules and practices <of federal agencies, anything spe'cif- 
ically exempt by Federal statute, things which are privileged in civil 

litigation, bank records, oil well data, and a variety of other types of 

' ' ■ • ^ • ' ' 1 ■ 

official records. J 

Investigs^tive reporters give a number of reasons why they con- 
tinue to seek (and- occasionally obtain illegally) pertain kinds of pirate 

records. They citti the public's right to know, the redress of criminal 

f 

or civil wrongs, or civil disobedience. While these reasons for the 
acquisition ^f records which are prohibited by law from being public are 
questionable even for journalists (i.e., they fall into a morally "grey" 
area which cannot be dismissed by refuge in a solid social conscience), 
they « are most surely unethical for the evaluator^. Nevertheless, there, 
will be "times when the acquisition of a document or record appears 
desirable, and 'it seems^^ be impossible to acquire it. The evaluatdr 

... I - ' 
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needs to balance carefully his sense of the public's right to know, the 
possible value of a record, and his willingness to engage in behavior 
which may be less than professional (e.g., asking an "insider" to 
obtain the record from the files for him). The lea^fif the Pentagon 
Papers— with the resultant break-in at the office of Daniel EUsberg's 
psychiatrist — are good examples of how records may be pbtained either 
from an inside. source (sometimes called by journalists a "kamikaze"), or 
irom clearly illegal means. The naturalistic evalijiator needs to ask- 
whether he wants, to be involved in either of those forms of record 
acquisition (even— and especially— if the records do not contribute to 
thp public's right to know as did the Pentagon Papers; the evaluator is 
simply not involved in those kinds of decisions on a' day-to-day basis). 
The journalists' stance is that there are no documents and records 
whidh .are inaccessible. Some just take more time or more devious 
means to procure than others. We would take the posture that that 
stance is simply nbt a ^ tolerable one for the kiijd of inquiry which 
attends most evaluative efforts. ' To the extent that it Ip clearly Illegal, 
it is off limits to the iiaturallstlc inquirer. Situations which are border- 
line must be dealt with ^ an individual basis, but the acquisition of 
records to which one has no legal access ought first to be- cleared with 
someone higher in authority be^re it Is undertaken. In most Instan- 
ces, serious professional or legal breaches In an effort to collect re- • 
cords is neither necessar^nor desirable. 

V . .. • . 

Epilog 

We have reviewed here some excellent techniques for making use of 
a variety of inexpensive afid rich resources — documents and records. 
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Both documentary analysis and record utilization are useful either on 
their own, that Is, as, a primary technique, or as supplementary tech- 
niques, using the concept of "multiple operation" research. With' new 
fiscal constraints on the amounts which may be spent for research, both 
techniques need to be explored as weapons in the methodological arsenal^ 
of naturalistic evaluator*. The strengths of the two techniques far 
outweight the costs (which tend to be solely in the collection pf such 
written material and the attendant analysis), and they can and do 
produce data which may be collected otherwise only by i^te costly 
means. , 

. The strengths of documentary analysis are manyfold. It is entire- 
ly unobtrusive and non-reactive. It results in data as well-grounded 
as any method, and documents themselves are unchanging, may be 
analyzed at any time,^^nd reanalyzed as often as needed, e.g., for 
reliability or auditor checks. Docimients (as opposed to records) ex- 
press the writer's perspectives in his or her own terms— a factor which 

» 

may be significant when personal or "natural" language is important. 
On the other hand, there are some weaknesses to the technique (which 
is why it is often used in conjunction with other methods). Documents 
are ^ften the result of some self-serving motive. They tend \p be 
uni-perspectival, to be spatially and temporally specific, and often 
represent unique and non-recurring events. Occasionally, documents 
fall to be representative, either because only a cectain sub-set is 
available, or becausie it is difficult to specify the populati^ which can 
then be subjected to the usual sampling rules. Documents are often 
difficult to subject to outside tests of \£0fiflfmability . And authors of 
documents may be biased or untruthful (although that is the case even 
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with paper-and-pencil tests, with Interviews, or wit^ other forms of 
-responses). 

There are problems also of source authenticity— whether the 
document is the actual one written by the author must be determined by 
& variety of credibility « checks ; problems of document credibility- 
questions of whether the information contained therein is based on 
primary or aecondak*y experie:;ice, whether it Is based on stenographic 
record, or taped, or simply a recollection, whether the author was 
cdfaipetent at the time he constructed the document, and whether it was 
intended in its present form for public consumption or thought to be 
private; and problems in the adequacy of the unitizing and categorizing 
process— are tl>e rules spelled out clearly enough so that subsequent 

» 

researchers can come to 4Jie same conclusions using those rules and the 
same set of docuAie;its? Thus, careful attention must be given in 
specific cases to capitalizing on strengths while taking possible weak- 
nesses into account « 

, )• ' 

The strengths of records occasionally overlap with those of docu- 
ments, in that they are unobtrusive and non-reactive. In addition, 
they are a rich and readily accessible resource (at least in most in- 
stances), and they are a low-cost resource. The reprefsent a natural 
or "in context" source,, of Information and a legally unassailable base" 
from which to defend onself against allegations of misinterpretation and 
libel. The primary problem with records utilization is simply finding 
out what records exist and where they may be had. This takes prac- 
tice, but freedom of information and sunshine laws make records more 
available than ever before, and ..they can and ought to be a useful 
primary or secondary tool for the naturalistic evaluator or inquirer. 

^ ■■ ^ . ■ 
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We haVe, in additon, discussed some of the more common techni- 

^ ques employed in .documentary and records analysis, namely content 

analysis ^nd case survey aggregation analysis (for documents) and 

cross'indexing ^nd Uhe . creation of chsonolpgies of events and .relation- 

ships^ (for records). The analysis of documents* and records, while 

. • • • 

contributing to multiple operations research, however also fosters the 
raising of ethical questions, particularly with resptct to their actual 
^ acquisition and individual rights to privacy. The inquirer must con- 
sider whether the acquisition of any giVen record will result in a public 
good so great that the record (or document) must be obtained at^^l|ly N» 
cost. Our posture on that issue has basically been conservative, 
erring on the side of avoiding that which is compromising or illegal. In 
any way. The rule of thumb might best be: "If in <ioubt (about how 
you may acquire this document without illegal means), don't." Most of 
the situations and events with which educations^ ^valuators must deal in 
everyday life simply are so^ public that the tactics of Investigative 
reporters in searching out crimes and violations of the public weal are 
not appropriate 4 

Finally, we urge the naturalistic Inquirer who has not had ex- 
perlesnce using these forms of Inforttiatlon to practice using them; to 
search fpr ways, to Incorporate analysis of records and documents into 
his own repertoire of methods; and use the methods to embed his own 
hypothesis-testing and theory generation with grounded data. 
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